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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 

James  B.  Greene  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 
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Small  Business  Research  in  a  World  of  Skewed  Returns 

Toby  Edison,  Defense  Acquisition  University 

Sean  F.  Crean — Mr.  Crean  is  the  director  of  the  Office  of  Small  Business  Programs  for  the 
Department  of  the  Navy.  He  serves  as  chief  advisor  to  the  Secretary  on  all  small  business  matters. 

He  is  responsible  for  small  business  acquisition  policy  and  strategic  initiatives. 

Mr.  Crean  joined  the  Secretary  of  the  Navy  Staff  as  a  member  of  the  Senior  Executive  Service  in 
January  2010  and  has  over  30  years  of  federal  service.  Prior  to  receiving  this  appointment,  he  served 
as  Deputy  Assistant  Secretary  of  the  Navy  for  Acquisition  and  Logistics  Management  during  a  two- 
year  military  recall  to  active  duty  as  a  rear  admiral  in  support  of  Operation  Iraqi  Freedom. 

Mr.  Crean’s  previous  experience  includes  serving  as  the  senior  procurement  analyst  for  the  U.S. 
Small  Business  Administration’s  Office  of  Government  Contracting  Area  I  (New  England)  for  19 
years.  In  this  role  he  was  the  principal  advisor  to  the  SBA’s  six  regional  district  offices  and 
congressional  delegations  on  procurement  issues.  He  provided  acquisition  strategy  analysis  for  over 
20  buying  activities  throughout  the  region,  supporting  both  DoD  and  civilian  federal  agencies.  He  first 
entered  federal  civilian  service  as  the  deputy  supply  officer  for  Naval  Air  Station  Brunswick,  ME, 
where  he  was  also  appointed  the  activity  small  business  specialist. 

Mr.  Crean’s  combined  military  and  civil  service  careers  have  provided  complimentary  and 
extensive  leadership  responsibilities  in  service  to  the  country.  As  a  member  of  the  reserve 
component,  he  has  attained  the  grade  of  rear  admiral  (two-star)  and  is  currently  assigned  as  deputy 
commander,  Naval  Supply  Systems  Command.  He  holds  a  Bachelor  of  Science  degree  in  business 
management  and  marine  transportation  from  State  University  of  New  York  Maritime  College  and  a 
Master  of  Business  Administration  degree  from  New  Hampshire  College’s  Graduate  School  of 
Business. 

He  has  a  number  of  personal  and  command  decorations,  including  two  Legion  of  Merit  awards. 
He  is  a  member  of  the  Defense  Acquisition  Corps  and  is  DAWIA  Level  III  Contracting  certified. 
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Thomas  Edison — Maj  Toby  Edison  is  a  professor  of  program  management  for  Defense  Acquisition 
University  (DAU),  West  Region,  Los  Angeles,  CA.  Edison  received  his  doctorate  at  the  RAND 
Graduate  School.  His  thesis  was  an  evaluation  of  the  DoD  Small  Business  Innovation  Research 
(SBIR)  program.  While  a  program  manager  for  Space  Radar  and  Joint  STARS,  he  initiated  several 
acquisition  innovations:  the  GMTI  Community  of  Practice  and  the  GMTI  Characterization  Lab.  The 
GMTI  Community  of  Practice  brings  together  stakeholders  to  improve  GMTI  capabilities  without  an 
established  program  of  record.  This  community  provided  guidance  on  GMTI  for  the  ISR  Task  Force. 
The  GMTI  Characterization  Lab  demonstrated  and  fielded  several  novel  GMTI  applications  and 
CONOPs.  He  is  on  active  duty  with  the  US  Air  Force  and  is  currently  deployed  overseas. 
[toby.edison@dau.mil] 

Abstract 

This  paper  observes  that  outcomes  from  small  business  support  programs  are  likely  to  have 
a  small  fraction  of  observations  with  extremely  positive  outcomes  that  may  impact  program 
evaluations  and  conclusions.  Current  evaluations  of  small  business  support  programs 
generally  estimate  the  efficacy  of  the  program  using  an  estimation  of  an  average  outcome  or 
an  average  treatment  effect.  Because  there  are  so  few  small  business  programs  that  have 
been  designed  for  evaluation,  researchers  often  report  a  simple  average  outcome  based  on  a 
survey  response  of  treated  firms  or,  in  very  rare  cases,  report  the  effectiveness  of  the 
program  as  an  average-treatment  effect  based  on  a  quasi-experimental  or  experimental 
design.  Unfortunately,  presenting  the  small  business  programs'  impact  as  an  average  might 
be  understating  the  magnitude  of  the  impact  of  the  program  because  the  actual  distribution  of 
the  outcome  of  the  program  is  probably  not  normally  distributed  and  is  probably  skewed  in 
the  positive  direction.  In  light  of  the  possibility  that  small  business  programs  produce  skewed 
outcomes,  this  paper  recommends  that  the  small  business  evaluations  include  discussions 
on  the  distribution  of  outcomes  to  better  inform  policy  makers,  administrators,  and 
participations  on  the  nature  of  the  outcomes. 

Introduction 

“Data  analysis,  like  calculations,  can  profit  from  repeated  starts  and  fresh 
approaches.  . . .  There  is  not  just  one  analysis  for  a  substantial  problem”  (T  ukey,  n.d.). 

This  paper  observes  that  outcomes  from  small  business  support  programs  are  likely 
to  have  a  small  fraction  of  observations  with  extremely  positive  outcomes  that  may  impact 
program  evaluations  and  conclusions.  Current  evaluations  of  small  business  support 
programs  generally  estimate  the  efficacy  of  the  program  using  an  estimation  of  an  average 
outcome  or  an  average  treatment  effect.  Because  there  are  so  few  small  business  programs 
that  have  been  designed  for  evaluation,  researchers  often  report  a  simple  average  outcome 
based  on  a  survey  response  of  treated  firms  or,  in  very  rare  cases,  report  the  effectiveness 
of  the  program  as  an  average-treatment  effect  based  on  a  quasi-experimental  or 
experimental  design.  Unfortunately,  presenting  the  small  business  programs’  impact  as  an 
average  might  be  understating  the  magnitude  of  the  impact  of  the  program  because  the 
actual  distribution  of  the  outcome  of  the  program  is  probably  not  normally  distributed  and  is 
probably  skewed  in  the  positive  direction.  In  light  of  the  possibility  that  small  business 
programs  produce  skewed  outcomes,  this  paper  recommends  that  the  small  business 
evaluations  include  discussions  on  the  distribution  of  outcomes  to  better  inform  policy 
makers,  administrators,  and  participations  on  the  nature  of  the  outcomes. 

This  paper  begins  with  a  review  of  small  business  evaluations  that  highlight  the 
universal  presentation  of  small  business  support  program  effects  as  averages.  Next,  the 
paper  focuses  on  observations  of  skewed  returns  in  related  research  fields:  technology 
policy  and  stock  market  returns.  The  paper  then  focuses  on  a  body  of  small  business 
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program  evaluations  that  highlight  the  possibility  that  outcomes  from  small  business  support 
programs  are  skewed  in  a  similar  nature  to  stock  market  returns  and  technology 
investments.  The  paper  concludes  with  empirical  and  non-empirical  strategies  for 
researchers  to  deal  with  the  skewed  nature  of  outcome  observations  to  better  craft  policy 
recommendations. 

If  a  Small  Business  Program  Evaluation  Reports  an  Outcome,  It  Is  an  Average 

This  section  of  the  paper  reviews  the  literature  on  small  business  support  program 
evaluations,  concluding  that,  generally,  if  an  effect  from  program  participation  is  estimated, 
that  effect  is  presented  as  an  average  effect.  The  findings  from  two  literature  reviews  on 
small  business  support  programs  are  presented.  The  first,  a  2008  working  paper  by  Gu, 
Karoly  and  Zissimopoulos  (2008),  summarizes  the  then-current  literature  on  small  business 
support  program  evaluations,  mostly  in  the  United  States.  The  second  review,  a  dissertation 
literature  review  chapter  from  2010  by  Edison  (2010),  summarizes  the  then-current  literature 
on  the  Small  Business  Innovation  Research  (SBIR)  program.  Both  document  that  most 
program  effects  are  presented  as  an  average  treatment  effect  or  as  an  average  of  survey 
responses. 

Small  businesses  are  believed  to  play  a  vital  role  in  the  economy,  generating  new 
jobs  and  fueling  innovation.  Because  of  this,  many  local,  state,  and  federal  small  business 
support  programs  have  been  created  to  finance,  incubate,  and  educate  entrepreneurs  in  the 
hopes  that  new  firms  will  create  new  jobs  and  increase  regional  prosperity.  Despite  the 
strong  support  and  billions  of  dollars  in  annual  funding  for  set-aside  contracts,  subsidized 
financing,  education,  and  business  zones,  there  is  relatively  little  rigorous  research 
conducted  on  small  businesses.  Moreover,  even  when  rigorous  research  is  conducted,  it 
often  reveals  very  little  information  about  how  and  why  programs  are  effective  (Gu  et  al. , 
2008). 

Gu  et  al.  (2008)  review  22  small  business  support  program  evaluations  that 
exclusively  report  program  effects  as  averages  of  the  outcomes  observed.  The  researchers 
documented  that,  of  the  22  studies,  20  of  the  studies  use  econometric  analysis  or  means 
comparison  methods,  with  two  studies  reporting  descriptive  summaries  of  surveys  of  the 
participants.  The  vast  majority  of  the  evaluations  were  published  without  comparing  the 
average  effect  with  treatment  groups.  Gu  et  al.’s  (2008)  review  documented  only  two  studies 
that  employed  experimental  methods;  Benus  (1994)  estimated  a  treatment  effect  from  a 
randomized  control  trial  that  offered  business  training  to  randomly  selected  groups  from  a 
population  of  individuals  interested  in  self-employment;  and  Bellotti  (2006),  which 
documented  the  experimental  design  Project  Growing  America  Through  Entrepreneurship 
(GATE),  estimated  a  treatment  effect.  Furthermore,  the  Gu  et  al.  (2008)  literature  review 
documented  only  two  econometric  quasi-experiments  that  used  matched  control  groups 
Lerner  (2006)  and  Sanders  (2002),  reinforcing  the  finding  that  very  few  small  business 
support  program  evaluations  use  rigorous  estimation  models. 

Edison  (2010)  reviewed  the  literature  that  documented  the  methods  used  in  the  39 
published  reports  that  evaluated  the  SBIR  program  from  1996  to  2010.  The  SBIR  is  a 
federal  research  and  development  (R&D)  program  that  mandates  that  large  federal 
agencies  set  aside  2.5%  of  their  external  R&D  budget  to  be  awarded  to  small  businesses 
through  the  SBIR  program.  The  GAO,  RAND,  the  Nation  Academies  of  Sciences,  and 
various  academic  researchers  have  extensively  studied  the  program.  Of  the  39  published 
evaluations  of  the  SBIR  program,  only  six  used  quasi-experimental  econometric  methods  to 
estimate  an  average  treatment  effect;  the  remainder  of  the  reports  presented  the  effects  of 
the  SBIR  program  with  a  simple  mean,  typically  from  a  self-reported  survey.  A  handful  of  the 
reports  used  other  qualitative  methods  to  describe  the  effects  of  the  program,  typically  case 
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studies  on  a  handful  of  participating  firms,  or  of  SBIR  program  administrators  in  the  various 
federal  agencies.  Regardless  of  the  level  or  rigor  employed  in  the  evaluations,  researchers 
unanimously  found  a  positive  average  (treatment)  effect  for  firms  that  participate  in  the 
program.  The  conclusion  to  a  2005  GAO  report  summarized  the  state  of  SBIR  program 
evaluations:  “an  issue  that  continues  to  remain  somewhat  unresolved  after  almost  two 
decades  of  program  implementation  is  how  to  assess  the  performance  of  the  SBIR  program” 

(P-1)- 

Despite  the  preponderance  of  average  effects  published  in  the  SBIR  literature  that 
Edison  (2010)  documented,  the  literature  review  documents  a  subset  of  analyses  that 
document  a  phenomenon  of  extreme  outliers  in  outcomes  in  Held  (2006). 1  The  most  obvious 
documentation  of  extreme  outliers  is  in  the  analysis  of  cohorts  of  first-time  Department  of 
Defense  (DoD)  contractors  who  won  their  first  DoD  contract  through  the  SBIR  program.  For 
these  cohorts  of  firms  (256  in  1995  and  220  in  1999),  non-SBIR  defense  contract  awards  for 
each  firm  were  observed  three  years  following  their  initial  SBIR  award.  In  each  cohort,  just 
three  firms  won  more  that  95%  of  the  entire  cohort’s  non-SBIR  DoD  contracts  (see  Figure 
1).  Edison  (2010)  observed  that  the  extreme  outlier  phenomenon  is  consistent  with  the 
Scherer  and  Harhoff  (2000)  generalization  that  returns  to  innovation  investment  appear  to 
be  concentrated  in  a  top  few  observations.  Scherer  and  Harhoff  (2000)  observed  the 
consistent  pattern  of  50-95%  of  returns  from  innovation  are  concentrated  in  the  top  10%  of 
study  population  observations;  they  further  observe  that  this  pattern  is  consistent  in  patents, 
Venture  Capital  investments,  IPOs,  startup  firms,  and  pharmaceutical  launches. 

RAND  Measures  of  DoD  SBIR  Goals: 

Increase  Private-Sector  Commercialization  of  Federal  R&D  (cont.) 


RAND  Y«,r 


Figure  1.  Graphical  Analysis  of  SBIR  “High  Flyers” 

(Held,  2006) 


1  The  author  of  this  article  was  also  a  member  of  the  Held  (2006)  research  team. 
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This  section  summarized  the  current  paradigm  of  evaluating  small  business  support 
programs  using  generally  accepted  econometric  analysis  and  mean  comparison  methods. 
Both  methods  have  strength  in  that  informed  laymen  and  advanced  researchers  easily 
understand  them.  Unfortunately,  both  methods  rely  on  the  assumption  of  normality  or 
“approximate  normality”  to  estimate  average  effects.  The  Edison  (2010)  observation  that  the 
rare  analysis  by  Held  (2006),  which  observed  effects  as  containing  a  few  high  performing 
outliers  (rather  than  an  average),  raised  doubts  about  the  “approximate  normality”  of  the 
outcome  variables,  thus  the  validity  of  using  averages  to  estimate  program  effects.  The  next 
section  further  discusses  Scherer  and  Harhoff’s  (2000)  seminal  observations  of  skewed 
returns  to  innovation  investments  and  presents  related  observations  on  skewed  returns  from 
the  financial  analysis  research  community  and  from  the  network  traffic  analysis  community. 

Exceptional  Observations  Literature  Review 

A  paradigm  of  research  agenda  focusing  on  skewed  returns  is  growing  in  the  diverse 
fields  of  innovation  policy,  financial  analysis,  and  network  traffic  analysis.  This  paradigm  shift 
has  been  levered  by  simultaneous  improvements  in  data  collection  and  data  processing  that 
have  enabled  researchers  to  collect  and  analyze  an  entire  census  of  population 
observations  rather  than  samples  of  the  population.  The  researchers  analyzed  the  “skewed- 
distributed  outcomes”  (Scherer  &  Harhoff,  2002)  or  “heavy  tailed”  distributions  (Mandelbrot, 
2003;  Resnick,  1997). 

Peck  and  Scherer’s  (n.d.)  research  on  skewed  returns  presented  evidence  from  eight 
data  sets  (see  Figure  1)  that  summarized  the  skewed-distributed  outcome  from  a  diverse 
range  of  innovation  investments  (e.g.,  patents,  pharmaceutical  introductions,  initial  public 
offerings  [IPOs]  of  firms,  and  firm  startups).  In  each  of  these  populations,  a  small  fraction  of 
observations  (-10%)  contained  a  majority  of  the  returns  of  the  entire  population.  The 
authors  documented  implications  that  this  generalization  has  for  policy  makers:  to  ensure 
success,  a  large  variety  of  projects  must  be  supported  rather  than  a  small  number  of 
national  champions;  and  to  measure  success,  policy  makers  must  be  able  to  support  a 
program  that  might  show  a  90%  failure  rate.  Scherer  and  Harhoff’s  (2002)  observations 
appear  to  have  been  incorporated  into  some  mainstream  research  and  development  policy 
evaluations  (Roessner,  2002). 

Scherer  and  Harhoff’s  (2002)  conclusion  that  a  small  minority  of  innovations  yields 
the  lion’s  share  of  all  innovations’  total  economic  value  also  has  implications  for  researchers, 
which  I  will  elaborate: 

•  If  researchers  are  using  sampling  methodology,  there  is  a  high  probability  that 
they  will  not  sample  some  of  the  most  significant  observations,  which  will 
dramatically  change  the  results  of  the  evaluation. 

•  Standard  econometric  analysis  and  mean  comparison  methods,  which  rely  on 
normality  assumptions,  might  not  be  valid. 

•  Lag  effects  and  spillover  effect  might  be  extremely  large  and  potentially  macro- 
economically  disruptive. 

•  Standard  sampling  methods  might  miss  the  few  most  important  observations  in 
the  population. 

•  Matching  algorithms  might  drop  key  observations. 

•  Analysis  of  the  returns  from  the  top  few  performers  might  be  sufficient  to  perform 
a  cost  benefit  analysis.  (Roessner,  2002) 
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Table  1.  Scherer  and  Harhoff  Returns  to  Innovation  Portfolios 

(Scherer  &  Harhoff,  2002) 


Proportion  of  innovation  samples’  total  value  realized  by  the  most 
valuable  10%  of  innovations 


Data  set 

Number  of 
observations 

Percent  of 
value  in  top  10% 

German  patents 

772 

84 

US  patents 

81-85 

Harvard  patents 

118 

84 

Six  university  patents 

1991  royalties 

350 

93 

1992  royalties 

408 

92 

1993  royalties 

466 

91.5 

1994  royalties 

411 

92 

Venture  Economics  startups 

383 

62 

Horseley-Keogh  startups 

670 

59 

Initial  public  stock  offerings 

110 

62 

(IPOs)  —  1995  stock  value 

Grabowski— Vernon 

1970s  drugs 

98 

55 

19S0s  drugs 

66 

48 

Limitations  of  Scherer  and  Harhoff’s  Skewed-Outcome  Observations 

In  a  real-world  sense,  Scherer  and  Harhoff  s  (2002)  observations  that  the  returns  to 
innovation  investments  have  a  long  skewed-outcome  tail  is  fundamental  for  policy  makers 
and  analysts  who  need  to  understand  the  nature  of  the  innovation  process.  Unfortunately, 
this  real-world  observation  has  a  significant  limitation  when  analyst  are  attempting  to  use 
standard  quantitative  tools  to  describe  what  they  are  seeing  in  the  data.  If  what  analyst  are 
seeing  can  be  characterized  as  having  a  finite  “thin  tail”  that  approximates  a  log-linear 
distribution,  then  researchers  can  use  standard  data  analytic  tools  by  transforming  the 
outcome  observations  with  a  log  function.  The  log-linear  functions  have  the  analytical 
strength  of  converging  to  stable  means  and  variances  with  large  numbers  of  observations. 
However,  if  what  analysts  are  seeing  is  an  infinite  and  or  “fat  tail”  distribution,  then  this 
distribution  is  better  described  with  a  Pareto-Levy  function,  which  has  the  analytically  difficult 
challenges  of  having  unstable  means  and  variances,  especially  when  larger  numbers  of 
observations  are  analyzed. 

Another  difficulty  both  log-normal  and  Pareto-Levy  distributions  have  is  that  they  are 
undefined  for  observations  with  zero  or  negative  values.  In  many  situations  when  analyzing 
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returns  to  innovation,  this  is  not  an  issue,  because  some  most  innovations  have  some  value 
greater  than  zero.  For  example,  in  the  case  of  IPOs,  the  stock  price  of  a  firm  represents  an 
approximation  of  the  value  of  assets  of  a  company,  which  will  probably  have  some  non¬ 
negative  worth.  However,  in  the  case  of  patents,  some  patents  might  not  have  any 
estimated  commercial  value.  This  limitation  is  even  more  worrisome  when  judging  the 
impacts  of  small  business  support  programs,  because  some  interventions  might  possibly 
yield  no  increase  in  future  sales,  employment,  or  earnings,  and  other  interventions  could 
actually  have  a  negative  impact  on  business  outcomes. 

A  careful  reading  of  Scherer  and  Harhoff  (2002)  reveals  a  worrisome  assumption: 
they  postulate  that  the  returns  to  innovation  “most  likely”  adhere  to  a  log-normal  law  (a  long, 
thin  tail)  but  open  the  possibility  that  their  research  “may  not  have  captured  the  most 
extreme  private  values.”  The  authors  leave  the  door  open  to  the  possibilities  of  skewed- 
outcomes  of  a  Pareto-Levy  distribution.  For  most  research  applications  with  a  small  number 
of  observations,  narrowly  defined  outcome  measures,  and  short-time  horizons,  the  log¬ 
normal  approximation  provides  researchers  and  policy  makers  with  useful  insights. 
Unfortunately,  the  models  employing  log-normal  assumptions  might  not  sufficiently 
represent  the  impacts  of  disruptive  innovations,  which  have  the  potential  to  have  significant 
macro-economic  impacts.  Disruptive  innovations  are  actually,  in  many  cases,  what 
innovation  and  small  business  support  programs  are  actually  seeking. 

In  aggregate,  the  literature  review  on  small  business  support  evaluation  in  the 
previous  section  of  this  paper  confirms  that  the  prevailing  paradigm  is  to  assume  normality 
and  present  an  average  effect;  given  this  current  paradigm  it  would  be  difficult  for 
researchers  to  publish  program  evaluations  that  abandoned  this  paradigm  to  make  it  past  a 
peer  reviewer  or  bureaucratic  sponsors  with  a  report  that  presented  their  results  as  a 
distribution  of  outcomes,  which  was  non-normal  and  skewed.  To  motivate  a  paradigm  shift  in 
small  business  research,  a  brief  introduction  to  the  current  paradigms  in  financial  research 
and  network  traffic  analysis  modeling  is  presented. 

How  Other  Disciplines  Embrace  Non-Normality 

Research  in  finance  and  telecommunications  network  modeling  has  long  embraced 
the  heavy-tailed,  skewed  distribution  phenomenon.  The  stylized  observations  or  non¬ 
normality  stemming  from  heavy-tailed  distributions,  have  caused  significant  re-examination 
of  the  analytical  foundations  of  these  disciplines. 

The  seminal  paper  on  non-normality  in  network  data  analysis  from  these  disciplines 
that  might  be  useful  for  small  business  support  policy  analysis  is  a  paper  by  Leland,  Taqqu, 
Willinger,  and  Wilson  (1994).  Leland  et  al.  (1994)  described  the  research  team’s  analysis  of 
Ethernet  LAN  traffic  as  “self-similar”  or  fractal.  This  observation  countered  the  then- 
prevailing  paradigm  of  analyzing  Ethernet  traffic  with  “Poisson-like”  models.  The  authors 
observed  that,  regardless  of  how  small  or  large  the  time  interval,  Ethernet  traffic  reveals  a 
pattern  of  “burstiness,”  which  contained  an  extremely  small  percentages  of  extraordinarily 
large  packages  of  information.  The  researchers  then  introduced  several  different  advanced 
statistical  methods  to  represent  the  behavior  of  the  data.  The  roots  of  these  analytical 
methods  are  found  in  the  body  of  work  by  Mandelbrot  (2003),  which  will  be  discussed 
shortly.  What  is  more  important  to  small  business  support  program  analysts  is  the  three 
implications  that  the  authors  recommended  for  engineering  on  how  to  design  systems. 

The  first  implication,  the  self-similar  nature  that  Ethernet  data  has  for  system  design, 
is  that  the  nature  of  the  sources,  destinations,  and  nature  of  the  individual  data  packets 
needs  to  be  analyzed.  Small  business  policy  researchers  can  certainly  appropriate  this 
observation  by  focusing  inside  the  “black-box”  of  select  firms  in  the  population  to  discover 
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the  nature  of  their  founding  story,  the  nature  of  the  market  they  are  selling  to,  and  their 
internal  structure.  The  second  observation  in  Leland  et  al.  (1994)  was  that  researchers  need 
to  account  for  and  describe  the  behavior  of  the  variation  in  network  traffic  as  longer  and 
longer  time  intervals  are  observed.  Because  each  population  under  analysis  is  necessarily  a 
finite  data-set,  a  mean  and  a  variance  can  always  be  calculated;  but  when  the  time  intervals 
are  increased,  the  mean  and  variance  might  not  converge  to  stable  estimates.  This 
observation  can  also  be  useful  to  small  business  policy  researchers,  who  should  describe 
the  effect  that  increasing  sample  size  or  time  intervals  have  on  the  mean  and  variance  of  the 
outcome  of  interest.  Finally,  Leland  et  al.  (1994)  detailed  the  implications  that  self-similar 
traffic  pattern  have  for  engineering  predictive  data  traffic  models.  With  better  knowledge  of 
the  behavior  of  the  traffic  data,  the  engineers  can  build  synthetic  models  of  proposed 
Ethernet  networks,  which  will  experience  better  traffic  flow,  less  congestion,  and  be  optimally 
sized  to  reduce  excess  capacity.  The  observation  that  better  system  design  is  possible  with 
a  better  understanding  of  the  nature  of  the  network  traffic  is  directly  applicable  to  small 
business  policy  researchers  in  better  describing  the  nature  of  the  observed  outcomes 
resulting  from  program  designs  so  that  the  delivery  of  the  small  business  support  can  be 
more  appropriately  tailored  to  providing  the  right  amount  of  support  for  small  businesses. 

This  seminal  paper  by  Leland  et  al.  (1994)  has  many  connections  to  Mandelbrot’s 
(2003)  research  on  non-normal  behavior  of  stock  market  returns,  portfolio  risk,  and  optimal 
portfolio  design.  The  Handbook  of  Heavy  Tailed  Distributions  in  Finance  edited  by  Rachev 
(2003)  contains  a  good  summary  of  the  research  and  theory  that  are  the  foundation  of  non- 
Gaussian  (non-normal)  financial  analysis.  Rachev  (2003)  offered  these  stylized  facts 
regarding  the  distribution  of  financial  data: 

•  non-Gaussian,  heavy-tailed  and  skewed  distributions, 

•  volatility  clustering, 

•  temporal  dependence  of  the  tail  behavior,  and 

•  short-  and  long-range  dependence. 

These  theories  regarding  the  behavior  of  financial  markets  are  directly  applicable  to 
small  business  policy  researchers  because  small  business  support  programs  inherently 
involve  financing  business  ventures  with  the  hopes  of  those  ventures  increasing  in  value 
and  improving  the  regional  or  national  economy. 

This  section  has  some  paradigm  shifting  observations  regarding  evaluations  of  small 
business  support  programs,  specifically  that  skewed-outcome  observations  can  invalidate 
normality  assumptions  and  therefore  call  into  question  the  recent  advancements  in  small 
business  policy  research,  which  has  made  significant  advances  to  estimate  average  effects. 
More  research  is  needed  to  determine  whether  the  log-normal  assumption  holds  that 
extreme  positive  outcomes  in  small  business  performance  can  be  characterized  as  a  thin  tail 
or  a  fat  tail.  It  is  the  intention  of  this  paper  to  motivate  future  researchers  to  detail  the 
distribution  of  the  outcomes  (especially  those  skewed  outcomes)  in  their  evaluations,  even  if 
average  effects  are  also  presented. 

A  Deeper  Example  of  Exceptional  Outcome  Data  on  Small  Business  Outcomes 

Edison  (2010)  detailed  the  distributions  of  pre-  and  post-treatment  outcomes  for  firms 
that  applied  for  DoD  SBIR  contracts  in  2003.  The  distributions  for  pre-treatment  non-SBIR 
DoD  contracts  clearly  adhere  to  the  skewed-outcome  distribution  Scherer  and  Harhoff 
(2002)  described  (see  Figure  3).  Likewise,  the  post-treatment  observations  (see  Table  2) 
illustrate  another  example  of  skewed,  long-tail  outcomes. 
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Given  the  previous  presentation  on  the  skewed  nature  of  outcomes  from  innovation 
investments  and  financial  markets,  it  seems  plausible  that  the  patterns  presented  from  this 
evaluation  of  the  DoD  SBIR  program  would  also  be  evident  in  other  evaluations  of  small 
business  programs.  The  only  way  to  verify  that  this  phenomenon  is  not  just  an  anomaly  is 
for  researchers  to  include  an  analysis  of  the  structure  of  the  data  sets  in  their  publications. 
Additionally  reviewers  of  research  on  small  businesses  should  also  request  that  their 
authors  include  graphical  and  statistical  summaries  of  the  skewed  nature  of  the  outcome 
variables.  Through  these  subtle  actions,  a  better  understanding  of  the  nature  of  the  impacts 
from  small  business  support  programs  can  be  characterized  and  help  improve  support 
programs. 
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Table  2.  Post-Treatment  Increase  in  DoD  Sales  (2003-2004) 


loser 

winner 

Observations 

773 

687 

Mean 

220,779 

649,822 

Stdev 

2,121,602 

4,468,850 

Max 

35,700,000 

61,600,000 

Min 

(7,620,000) 

(23,800,000) 

%  Negative 

12% 

18% 

%  Positive 

15% 

29% 

%  zero 

73% 

53% 

%  extreme  neg  (<2stdev) 

0.39% 

0.29% 

%  extreme  pos  (<2stdev) 

1.29% 

2.04% 

%  extreme 

1.68% 

2.33% 

Recommendations  for  Researchers 

1 .  Present  the  distribution  of  the  outcome  data  with  both  normality  (or  log- 
normality)  assumptions  and  non-normality  assumption. 

2.  Present  the  average  outcome  (from  the  survey)  or  average  treatment  effect 
(from  the  experiment  or  quasi-experiment). 

3.  Do  case  studies  on  average  firm,  well-below-average  firm,  and  exceptional 
firm. 

4.  Unleash  the  creative  forces  of  policy  researchers  to  invent  a  way  to  perform 
program  evaluations  in  a  world  of  skewed  returns. 

Peck  and  Scherer  have  noted  in  numerous  studies  that  returns  to  innovation  are 
highly  skewed.  This  paper  surveys  literature  on  small  business  evaluation  programs,  finding 
that  most  researchers  estimate  an  average  treatment  effect.  This  paper  also  reviews  a 
subset  of  the  small  business  literature  on  DoD  small  business  programs  to  conclude  that  the 
returns  to  these  programs  appear  to  be  skewed  as  well.  Researchers  and  policy  makers 
should  consider  the  possibility  that  returns  to  small  business  support  programs  might  be 
skewed;  therefore,  in  evaluating  the  programs,  a  small  average  treatment  effect  might  be 
hiding  a  large  effect  concentrated  in  a  few  high  return  successes. 

Policy  makers  considering  small  business  interventions  should  be  aware  of  the 
phenomenon  of  skewed  returns  in  building  their  support  interventions;  and  in  the  rare 
instance  that  the  policy  makers  choose  to  evaluate  the  intervention,  they  should  consider 
augmenting  average  treatment  effect  with  more  focused  case  studies  on  the  high  flyers. 
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